Feat: kill and purge for HTCondor - #36

…` removed

… have them link to each other

…` removed

…ede into feat/core-contract

…a consumer should choose to use it or not

* feat: Models the core contract needs

* Feat: Protocols and Exceptions added for the core contract

* Feat: Private protocols for the BatchBackend

* feat: a fake backend to run unit tests against

* fix: some code cleanup in the helper functions

* feat: setup for the unit test suite

* chore: style fix

* chore: styling, unit test fixture

* chore: single letter var `h` refering to hanlde changed to `handle`

* fix: unused parameter in `test_injected_failure_is_reported_per_job()` removed

* feat: Models the core contract needs

* Feat: Protocols and Exceptions added for the core contract

* Feat: Private protocols for the BatchBackend

* feat: a fake backend to run unit tests against

* fix: some code cleanup in the helper functions

* feat: setup for the unit test suite

* chore: style fix

* chore: styling, unit test fixture

* chore: single letter var `h` refering to hanlde changed to `handle`

* fix: unused parameter in `test_injected_failure_is_reported_per_job()` removed

* fix: made the OutputSpec optional as it is not suppoerted by all and a consumer should choose to use it or not

---------

Co-authored-by: alexander <alexander.auer@cern.ch>

…expansion

Two tests had moved without changing, which only made the diff harder to read.

…merged as DIRACGrid#28)

# Conflicts:
#	src/intercede/_backends/fake.py
#	src/intercede/registry.py
#	tests/unit/conftest.py
#	tests/unit/test_registry.py

…, squash-merged as DIRACGrid#29)

A value given by mistake, such as a misspelled field or plain data in a
CredentialSet, may be the secret itself. Validation errors now hide their
input, as the configuration models already do.

The message comes from the consumer's code and may hold a token or a
secret URL. The AuthenticationError names only its type; the exception
itself is still available as __cause__.

The fields were described in comments, which the generated reference does
not show. They now use attribute docstrings, as the configuration models do.

Name every argument and the return value, and drop the blank lines
before closing quotes.

For one of several CEs on a host, or a CE behind a forwarded address. The
endpoint and schedd end up on the tools' command lines, so they are checked
when the configuration is read: neither can pass as an option.

Part of DIRACGrid#23.

The contract says inputs are read before submit returns, so an unreadable
one is a SpecificationRejectedError, not a raw OSError. The conformance suite
now asks for it, and the fake does it.

Part of DIRACGrid#23.

Moves test_fake_backend.py to tests/unit/backends/fake/, ready for the
HTCondor tests that follow. The tests themselves do not change.

Part of DIRACGrid#23.

- process: runs one tool, killing it on a timeout
- session: one operation's credentials in private files that exist only
  while it runs, the tools' environment, and telling a rejected token from
  an unreachable CE
- staging: the private working area, and what a submission puts in it

Nothing uses these yet. The backend follows in the next commit.

Part of DIRACGrid#23.

Renders a submit description from a plan: one process per copy, per-copy
values indexed by ProcId, quoting, and refusing what a description cannot
hold. It sets no policy on the job except keeping finished jobs until they
are purged, so output can be collected again.

Shared with the HTCondor scheduler to come (DIRACGrid#21), so it knows nothing about
any backend. Expected values were checked against HTCondor 25.

Part of DIRACGrid#23.

Spooled submission to a remote schedd, token authentication, and an optional
X.509 proxy for sites that account on VOMS attributes. Copies whose variable
names differ get separate clusters, because HTCondor merges each process's
environment onto the previous one.

Registered as `htcondor-ce`. Kill, purge and output follow in their own
branches.

Part of DIRACGrid#23.

Kill only removes active jobs, because removing a finished one would release
output that has not been collected. That is what purge is for. A job removed
a moment ago is still queued but already released, so purging twice is
harmless. Token scopes follow the operation: compute.cancel here.

Part of DIRACGrid#23.

Builds the tools' command lines and reads what they print: condor_submit
-terse, condor_rm, and the job ads of condor_q and condor_history, mapped
to the common JobStatus.

A finished job's result comes from its exit, a removed job keeps the result
it had, and any hold that interCEde did not cause counts as aborted.

Part of DIRACGrid#23.

FakeCondor answers the way HTCondor 25 does (checked against a live schedd),
so these run without HTCondor installed: credentials, error classification,
staging, secrets, containers and the working area.

Part of DIRACGrid#23.

Whether a purged job's status is then UNKNOWN or a final state depends on
whether the resource still remembers it: a schedd's history does, an in-memory
fake does not. Both are within the contract.

Part of DIRACGrid#23.

interCEde no longer supports proxies, so a description has nothing to say about one.

…ack/htcondor-3-submit-status

# Conflicts:
#	src/intercede/config.py

The credential provider now returns one credential instead of a set, and proxies
are gone: no proxy file, and no delegation setting to go with it. The kind is
checked once, where the token comes from, so the rest only deals with a token.

X509_CERT_DIR stays. It holds the CAs the CE's host certificate is checked against,
which has nothing to do with a proxy.

…ill-purge